The following definitions are used in this document:
“Securing” implies both encryption of data in transit as well as authenticating that the data has not been manipulated in transit.
A “secure tunnel” between two devices ensures that data passing between the devices is secure.
The “security polices” or “policies” for a secure tunnel define the traffic to be secured by source and destination IP address, port, and/or protocol. They also define the type of security to be performed.
A “key” for a secure tunnel is the secret information used to encrypt and decrypt (or authenticate and verify) the data in one direction of traffic in the secure tunnel.
A “Policy Enforcement Point” (PEP) is a device that secures the data based on the policy.
Existing Network Security Technology
According to the most commonly used computer networking protocols, network traffic is normally sent unsecured without encryption or strong authentication of the sender and receiver. This allows the traffic to be intercepted, inspected, modified, or redirected. As a result, either the sender or receiver can falsify their identity. In order to allow private traffic to be sent in a secure manner, a number of security schemes have been proposed and are in use. Some are application dependent, as with a specific program performing password authentication. Others, such as Transport Layer Security (TLS), are designed to provide comprehensive transport layer security such as the HTTP (web) and FTP (File Transfer Protocol) level.
Internet Security (IPsec) was developed to address a broader security need. As the majority of network traffic today is over Internet Protocol (IP), IPsec was designed to provide encryption and authentication services to this traffic regardless of the application or transport layer protocol. This is done, in IPsec tunnel mode, by encrypting a data packet (if encryption is required), performing a secure hash (authentication) on the packet, then wrapping the resulting packet in a new IP packet indicating it has been secured using IPsec.
The secret keys and other configuration data required for this secure tunnel must be exchanged by the parties involved to allow IPsec to work. This is typically done using Internet Key Exchange, IKE. IKE key exchange is done in two phases.
In a first phase (IKE Phase 1), a connection between two parties is started in the clear. Using public key cryptographic mechanisms, where two parties can agree on a secret key by exchanging public data without a third party being able to determine the key, each party can determine a secret for use in the negotiation. Public key cryptography requires each party either share secret information (pre-shared key) or exchange public keys for which they retain a private, matching, key. This is normally done with certificates (Public Key Infrastructure or PKI). Either of these methods authenticates the identity of the peer to some degree.
Once a secret has been agreed upon in IKE Phase 1, a second phase (IKE Phase 2) can begin where the specific secret and cryptographic parameters of a specific tunnel are developed. All traffic in phase 2 negotiations are encrypted by the secret from phase 1. When these negotiations are complete, a set of secrets and parameters for security have been agreed upon by the two parties and IPsec secured traffic can commence.
When a packet is detected at a Security Gateway (SGW) with a source/destination pair that requires IPsec protection, the secret and other security association (SA) information are determined based on the Security Policy Database (SPD) and IPsec encryption and authentication is performed. The packet is then directed to an SGW that can perform decryption. At the receiving SGW, the IPsec packet is detected, and its security parameters are determined by a Security Packet Index (SPI) in the outer header. This is associated with the SA, and the secrets are found for decryption and authentication. If the resulting packet matches the policy, it is forwarded to the original recipient.
General Limitations of IPsec
Although IPsec tunnel mode has been used effectively in securing direct data links and small collections of gateways into networks, a number of practical limitations have acted as a barrier to more complete acceptance of IPsec as a primary security solution throughout industry.    Configuration of Policies—Each SGW must be configured with each pair of source and destination IP addresses or subnets that must be secured (or allowed in the clear or dropped). If there are 11 SGW units fully meshed, each protecting 10 subnets, this requires 1000 policies in the SPD. This is a challenge in terms of the user setting up the policies, the time required to load the policies, the memory and speed difficulties in implementing the policies, and the increase in network time spent performing negotiations and rekey. The time for initial IKE negotiations in this example might be 10 minutes or more.
In addition, even for smaller networks, it requires the user to have a complete knowledge of all protected subnets and their security requirements. Any additions or modifications must be implemented at each gateway.    Certificate/PKI Management—PKI can become complex and difficult to manage. At minimum, it is intimidating to many network managers. However, strong PKI implementation is at the heart of effective security using IPsec (or TLS for that matter). The SGW should make this aspect as easy as possible for the network manager.    Multicast/Broadcast Traffic—IPsec in its present configuration cannot secure multicast or broadcast traffic. This is because keys are established between two entities and multicast or broadcast involves sending traffic from one source to many destinations at once.The Internet Engineering Task Force (IETF) has a couple of Requests for Comments (RFCs) in place or in process to address group domain of interpretation (GDOI), or group secure association key management protocol (GSAKMP). GDOI is generally available, for example, on Cisco devices.    Load Balancing—Many large network implementations require load balancing or other Quality of Service (QOS) techniques where traffic to a particular address may take one of a number of paths. If a set of SGW units must be placed along these parallel paths, there might be no way to assure which SGW traffic sees. As IKE provides secrets only between a pair of SGW units (remote and local), traffic to the second SGW would require a different set of secrets. In the existing IPsec implementations, this is impossible. The result is a limitation in the placement of SGW units in the network which may not be possible in certain situations.    Network Address Translation (NAT)—There are various forms of NAT, all of which cause problems for IPsec.
With Static NAT, a source IP address on an outgoing packet is replaced with an assigned replacement IP address. If the SGW exists before the static NAT device, the original source IP address will still exist in the encrypted packet and will be exposed on decryption. This would likely create problems on the receiving network or on the return packet. Dynamic NAT (which is rarely used) is similar except that the replacement IP address comes from an available pool. In either case, the SGW must be placed outside the NAT device.
In masquerading dynamic NAT (NAPT), the source IP address of a packet is replaced with a new source IP address and the port number is changed to identify the original source IP address and port. This might be done to provide a single IP address to the wide area network (WAN) for a large number of IP addresses in the local area network (LAN).
Unfortunately, if the SGW is behind the NAT device, IPsec hides the port and IP address on the original packet and does not provide a port on the outer header. The NAPT protocol is broken without a port to modify. A mechanism called NAT-Traversal (NAT-T) had been added to IPsec to address this problem. This can also be addressed by placing the SGW outside the NAT devices. Normally this cannot be done in cases of remote access by a home user running the IPsec gateway on their computer.
Further variations of NAT can be combined with load balancing, creating virtual servers, or providing QOS which combines the problems of NAT with the load balancing problem described above.    Firewalls/Intrusion Detection Systems (IDS)—A firewall or IDS can create conflict with IPsec as they may require inspection of the packet beyond the outer header (Layer 3). Firewall rules are often set to manage connections based on port or protocol, but this information is stored in the encrypted packet under IPsec. An IDS normally does deep packet inspection for viruses, worms, and other intrusion threats. Again, this information is encrypted under IPsec. Many firewall functions can be implemented using well written IPsec policies, although this can complicate the SPD entries. If the SGW is on the WAN side of the firewall and IDS, this problem is eliminated.    Path Maximum Transfer Unit (PMTU) and Fragmentation—The PMTU specifies the maximum IP packet size that can be sent. Above that size, packets must be fragmented to be sent in smaller sizes. A protocol for PMTU discovery permits a device to send larger and larger packets with a Do Not Fragment bit set. This continues until a device with a path limitation sends back a message that the packet is too large. Other networks simply set the PMTU to a specific value.
In IPsec, however, the packet is made larger by the IPsec header information. If the devices behind the SGW uses the largest packet size, the SGW must either fragment the packet, which can be slow and certainly reduces network efficiency, or ignore the PMTU. To avoid this problem, networks must employ PMTU discovery or set the PMTU for devices behind the SGW smaller than for the main network.    Resilient Network Traffic—If the network is implementing resiliency, it will likely require the secure solution be resilient as well. This can be accomplished with a virtual router redundancy protocol (VRRP), but a switchover would result in the need to rekey all traffic. In a fully meshed situation, this could be a significant interruption. If fast switchover is required, a resilient gateway with shared state may be needed.
In addition, one of the most significant barriers to general acceptance of IPsec as a security solution is the challenge of securing the data as it leaves on computer to where it enters the remote computer. This level of security, combined with authentication and authorization on each side, would extend security from just covering the WAN (e.g., the internet) to protecting data from unauthorized internal access. Some of the general limitations of IPsec are exacerbated by end-to-end deployment. For example, the IPsec implementation cannot be place on the WAN side of the firewall, IDS, NAT device, or any load balancing between virtual servers. There are a number of hurdles to true end-to-end security in addition to the general limitations described above:    Installation of an IPsec/IKE Stack on Individual PCs—With the variety of available operating systems (Windows XP, XP Service Pack 1 and 2, Linux and all it's kernel releases, etc.) and hardware platforms, a software implementation of the IPsec stack, which is dependent on both of these, must be designed, compiled, tested, and supported for each implementation.
Hardware solutions, such as IPsec on a NIC, provide some separation from these issues, but preclude automated remote installation of the IPsec stack.
In addition, the computer with the installation must be configured with the user certificate and the policy configuration. Ideally, the user would be identified in some way other than a machine based certificate. Unfortunately, all existing implementations require the computer to be configured directly, normally by a network security manager. IKE offers methods for remote access using certificate based authentication combined with RADIUS and XAUTH for the user ID as well as mode configuration to supply the user with a local network identification.    Limitation in Ability to Provide High-Speed, Low Latency, and High Number of SAs and Policies—A software solution on a computer (or mobile device) would be unable to provide high speed encryption or latency as low as on the existing SGW. In some cases this doesn't matter, but in situations with a high speed connection or involving streaming data, this may be significant. A hardware solution may suffer this limitation as well due to heat, space, or power considerations.
Either solution may be limited in the number of SAs or policies that are supported. This could be critical in a large, meshed security situation.